I ve made the Testtools committers team own both the project and the trunk branch for both pyjunitxml and testscenarios. This removes me as a SPOF if anything needs doing in those projects any Testtools committer can now do it. (Including code review and landing). If you are a testtools committer and need PyPI release rights, ping me and I ll add you. (I wish PyPI had group management).

I ve recently caught up on a bunch of reading some of which are worth commending.

• Switch documents the factors that cause changes to fail (both in organisations and personal stuff), and provides a recipe for ensuring you have addressed those factors in any change you are planning.
• The Lean Startup Applies Lean principles to the learning what customers respond well to in the same way that Lean removes waste from the process of building some X, this removes waste from the process of determining what that X should be.
• The Innovator s Solution Pop science report of research done on why disruptive innovation at existing companies fails; covers structure, management, funding, market analysis, has recommendations to remove these sure-fail cases.
• The Innovator s DNA Pop science report of research done into how people innovate : turns out that there are a lot of things that one can do to be a better innovator.

Read them all, or none. I enjoyed them all.

This is a tiny PSA prompted by my digging into a deadlock condition in the Launchpad application servers. We were observing a small number of servers stopping cold when we did log rotation, with no particularly rhyme or reason. tl;dr: do not call any non-reentrant code from a Python signal handler. This includes the signal handler itself, queueing tools, multiprocessing, anything with locks (including RLock). Tracking this down I found we were using an RLock from within the signal handler (via a library ) so I filed a bug upstream: http://bugs.python.org/issue13697 Some quick background: when a signal is received by Python, the VM sets a status flag saying that signal X has been received and returns. The next chance that thread 0 gets to run bytecode, (and its always thread 0) the signal handler in Python itself runs. For builtin handlers this is pretty safe e.g. for SIGINT a KeyboardInterrupt is raised. For custom signal handlers, the current frame is pushed and a new stack frame created, which is used to execute the signal handler. Now this means that the previous frame has been interrupted without regard for your code: it might be part way through evaluating a multi-condition if statement, or between receiving the result of a function and storing it in a variable. Its just suspended. If the code you call somehow ends up calling that suspended function (or other methods on the same object, or variations on this theme), there is no guarantee about the state of the object; it becomes very hard to reason about. Consider, for instance, a writelines() call, which you might think is safe. If the internal implementation is for line in lines: foo.write(line) , then a signal handler which also calls writelines, could have what it outputs appear between any two of the lines in writelines. True reentrancy is a step up from multithreading in terms of nastiness, primarily because guarding against it is very hard: a non-reentrant lock around the area needing guarding will force either a deadlock, or an exception from your reentered code; a reentrant lock around it will provide no protection. Both of these things apply because the reentering occurs within the same thread kindof like a generator but without any control or influence on what happens. Safe things to do are:

• Calling code which is threadsafe and only other threads will be concurrently calling.
• Performing atomic (any C function is atomic as far as signal handling in Python is concerned) operations such as list.append, or foo = 1 . (Note the use of a constant: anything obtained by reading is able to be subject to reentrancy races [unless you take care ])

In Launchpad s case, we will be setting a flag variable unconditionally from the signal handler, and the next log write that occurs will lock out other writers, consult the flag, and if needed do a rotation, resetting the flag. Writes after the rotation signal, which don t see the new flag, would be ok. This is the only possible race, if a write to the variable isn t seen by an in-progress or other-thread log write. That is all.

While some folk look down on fakeraid (that is BIOS based RAID-until-OS-takes-over) solutions, I think they are pretty neat: they let a user get many of the benefits of dedicated controller cards at a fraction of the cost. The benefits include the usual ones for RAID more spindles to handle IO, tolerance of disk failures. And unlike pure LVM solutions, you can boot from a degraded RAID 1 / 5 / 10 set because the BIOS knows how. In some ways this is better than dedicated cards, because we have the software take over, so we can change the algorithms for IO dispatch all the way down to the individual devices However, these RAID volumes are in a pretty awkward spot for installers and bootloaders: inside a running Linux environment they look like software RAID which cannot be depended on for booting, but at boot time they look like hard disks which cannot be looked under the hood. I recently got a new desktop machine which has one of these motherboards, and fortuitously my old desktop I was replacing had the same size disks so I had 4 disks and the option of using a RAID setup. Apparently I m a sucker for punishment because I went for a RAID 10 (that is two RAID volumes made up of two-disk mirrors (the RAID 1 component), and then those two volumes are combined via striping (the RAID 0 component). This has the potential for pretty nice performance: in principle any read can come from one of 2 disks, and every 64KB (the stripe size) of linear data will switch to the other mirror set, giving a nice boost. Writes need to write to 2 disks always, but every 64KB worth of data will alternate mirror sets, also giving a boost. Sadly we (Ubuntu) aren t ready for this yet: there are two key bugs that make this layout almost impossible to install into. This blog post is for my exo-memory, I want to be able to figure out what I did next time around . Firstly parted_devices, a helper used by Ubiquity and debian-installer to determine which block devices are actually disk drives that one can partition and install onto, has a confused heuristic when dealing with dmraid it looks for devices which are not layered on other dmraid devices. This handily excludes partitions, but has the undesirable effect of excluding that striped device because it is layered on the two mirrored devices. Bug 560748 was filed about that, and I ve added a workaround to it basically disabling the filtering, so its not suitable as a long term fix, but it will let one select the RAID volume correctly. Secondly, grub2, which needs to figure out what the name at boot time of the RAID volume will be currently gets confused. I don t know enough to really explain and be correct in my explanation but I do have a fugly patch which worked for me. Bug 803658 tracks this defect. The basic approach I took was to say that dmraid devices should be an abstraction layer we don t peek under: if it claims to be a disk, well then its a disk. As grub does actually work that way - it talks to INT 13h the BIOS support for booting off of the RAID volume is entirely sufficient. Sadly neither bug is at the point where the patches can be rolled into Ubuntu itself, but the workaround should let folk get up and running. In both cases, build the package locally in the installer, install it, then after than run ubiquity and things should install. After the install, you will need to reapply the patch in the resulting installed environment, or things like update-grub will die on you! (huge thanks to cjwatson and ev for giving me some tips while I investigated this)

Ok, so micro rant time: this is the effect of not taking things upstream: hardware doesn t work Out Of The Box. Very briefly, I purchased a Vodafone prepaid mobile broadband package today, which comes with a modem and SIM. The modem is a K3571-Z, and Ubuntu *thinks* it knows how they work (it doesn t). So it fails to connect in NetworkManager with a rather opaque NO CARRIER message. Thanks to excellent assistance from Matt Trudel, we tracked this down to a theory that perhaps modemmanager is using the wrong serial port and voila, it is. From there, the config file (/lib/udev/rules.d/77-mm-zte-port-types.rules) was an obvious next step and indeed there is no entry in there for the 19d2:1010 the K3571-Z. Google found one immediately though, on a Vodafone research site. The awful shame is this: that was committed to the bcm project in March this year. If Vodafone had shipped off a patch to modemmanager, we could have had that in 10.10, and possibly even in 10.04. There are plenty of users having trouble on Whirlpool etc with this model who would have had a better experience helping Vodafone s users be happier. All it would have taken is an email I m sure Vodafone want a great experience for their users, but I think they re failing to separate out platform improvements share and share alike, and branding / custom facilities. The net impact is harmful, not helpful. Anyhow, Natty will support this modem.

Tesetrepository has a really nice workflow for fixing a set of failing tests:

1. Tell it about the failing tests (e.g. by doing a full test run, or running a single known failing test)
2. Run just the known failing tests (testr run failing)
3. Make a change
4. Goto step 2

As you fix up the tests testr will just give your test runner a smaller and smaller list of tests to run. However I haven t been able to use that feature when developing (most) Python programs. Today though, I added the necessary support to testtools, and as a result subunit (which inherits its thin test runner shim from testtools) now supports load-list. With this a simple .testr.conf can support this lovely workflow. This is the one used in testrepository itself: it runs the testrepository tests, which are regular unittest tests, using subunit.run this gives it subunit output, and tells testrepository how to run a subset of tests.

[DEFAULT]
test_command=python -m subunit.run $IDOPTION testrepository.tests.test_suite
test_id_option=--load-list $IDFILE


So a while back I blogged about maintainable test suites. One of the things I ve been doing since is fiddling with the heart of the fixtures concept. To refresh your memory, I m defining fixture as some basic state you want to reach as part of doing a test. For instance, when you ve mocked out 2 system calls in preparation for some test code that represent a state you want to reach. When you ve loaded sample data into a database before running the actual code you want to make assertions about that also represents a state you want to reach. So does simply combining three or four objects so you can run some code. Now, there are existing frameworks in python for this sort of thing. testresources and testscenarios both go some way towards this (and I and to blame for them ), so does the zope testrunner with layers, and the testfixtures project has some lovely stuff as well. And this is without even mentioning py.test! There are a few things that you need from the point of view of running a test and establishing this state:

• You need to be able to describe the state (e.g. using python code) that you wish to achieve.
• The test framework needs to be able to put that state into place when running the test. (And not before because that might interfere with other tests)
• And the state needs to be able to be cleaned up.

Large test suites or test suites dealing with various sorts of external facilities will also often want to optimise this process and put the same state into place for many tests. The (and I m not exaggerating) terrible setUpClass and setUpModule and other similar helpers are often abused for this. Why are they terrible? They are terrible because they are fragile; there is no (defined in the contract) way to check that the state is valid for the next test, and its common to see false passes and false failures in tests using setUpClass and similar. So we also need some way to reuse such expensive things while still having a way to check that test isolation hasn t been compromised. Having looked around, I ve come to the conclusion we ll all benefit if there is a single core protocol for doing these things, something that can be used and built on in many different ways for many different purposes. There was nothing (that I found) that actually met all these requires and was also tasteful enough that folk might really like using it. I give you fixtures . Or on Launchpad. This small API is intended to be a common contract that all sorts of different higher level test libraries can build on. As such it has little to no policy or syntatic sugar. It does have a nice core, integration with pyunit.TestCase, and I m going to add a library of useful generic fixtures (like temporary directories, environment isolators and so on) to it. I d be delighted to add more committers to the project, and intend to have it be both Python 2.x and 3.x compatible (if its not already my CI machine isn t back online after the move yet, I m short of round tuits). Now, if you re writing some code like:

class MyTest(TestCase):
    def setUp(self):
        foo = Foo()
        bar = Bar()
        self.quux = Quux(Foo(), Bar())
        self.addCleanup(self.quux.done)

You can make it reusable across your code base simply by moving it into a fixture like this:

class QuuxFixture(fixtures.Fixture):
    def setUp(self):
        foo = Foo()
        bar = Bar()
        self.quux = Quux(Foo(), Bar())
        self.addCleanup(self.quux.done)
class MyTest(TestCase, fixtures.TestWithFixtures):
    def setUp(self):
        self.useFixture(QuuxFixture)

I do hope that the major frameworks (nose, py.test, unittest2, twisted) will include the useFixture glue themselves shortly; I will offer it as a patch to the code after giving it some time to settle. Further possibilities include declared fixtures for tests, and we should be able to make setUpClass better by letting fixtures installed during it get reset between tests.

I recently moved withing Canonical from being a paid developer of Bazaar to take on a larger challenge Technical Architect for Launchpad. Its been two months now, and its time to put my head up out of the coal face, have a look around and regroup. When I worked on Bazaar, every day when I started work got up I was working on a tool anyone can use, designed for collaboration upon sourcecode, for people writing software. This is a toolchain component right at the heart of the free software world. Bazaar and tools like it get used everyday to manage, distribute and collaborate on the sourcecode that makes up the components of Ubuntu, Debian, Fedora and so forth. Every time someone new starts using Bazaar for a new free or open source project, well I felt happy happy that in my small part I m helping with this revolution we re carrying out. Launchpad is pretty similar to Bazaar in some ways. Obviously they are both free software, both are written in Python, and both are sponsored by Canonical, my employer. And they both are designed to assist in collaboration and communication between free software developers albeit in rather different ways. Bazaar is a tool anyone can install locally, run as a command line, GUI, or local webserver, and share code either centrally (e.g. by pushing to Launchpad), or in a peer to peer fashion, acting as their own server. Launchpad, by contrast is a website which (usually) folk will use as a service in their browser, from the comand line FTP (for package building), ssh (for Bazaar branch pushing or pulling), or even local GUI programs using the Launchpad API service. This makes it more approachable for first time collaborators, but its less able to be used offline, and it has all the usual caveats of web sites : it needs a username and password, it s availability depends on the operators on the team I m part of. So there s a lot less room for error: if we do something wrong, the system is unavailable, and users can t just apt-get install an older release. With Launchpad our goal is to to get all the infrastructure that open source need out of the way, so that they can focus on their code, collaboration within their team and almost uniquely collaboration with other teams. As well as being open source, Launchpad is free for all open source projects to use Ubuntu is our single biggest user they use it for all bugtracking, translation and package building, and have a hugefraction of the total storage overhead in the database. Launchpad is a pretty nice system, so people use it, and as a result (on a technical basis) it is suffering from its own success: small corner cases in the code turn up every day or two, code written years ago to deal with a relatively small data set now has to deal with data sets a thousand or more times larger (one table, for instance, has over 600,000,000 rows in it. For the last two months then, I ve been working on Launchpad. As Technical Architect, I need to ensure that the things that we (users, stakeholders and developers of Launchpad) want to do are supported by the structure of the system : the platform(s) we re building on, the way we approach problems, coding standards and diagnostic tools. That sounds pretty dry and hands off, but I m finding its actually very balanced. I wrote a presentation when I started the job, which encapsulated the challenges I saw in front of the team on this purely technical front, and what I thought I needed to do. I think I was about right in my expectations: On a typical day, I ll be hands on in a problem helping get it diagnosed, talking long term structural changes with someone around how to make things more efficient / flexible / maintainable, and writing a small patch here or there to help move things along. In the two months since I took on this challenge, we ve made significant headway on the problem of performance for Launchpad : many inefficient code paths have been identified and removed, some new infrastructure has been created as is being rolled out to make individual pages faster, and we ve massively increased the diagnostic data we get when things go wrong. We ve introduced facilities for responding more rapidly to issues in the software (but they have to be rolled out across the system) and I hope, over the next 4 months we ll reach the first of my performance goals: for any webpage in Launchpad, it will complete rendering in 99% of the time. (Note that we already meet this goal if you measure the whole system, but this is biased by some pages being very frequently hit and also being very small).

No, not the bzr switch rather the book : http://www.amazon.com/Switch-Change-Things-When-Hard/dp/0385528752 Read it, its solid.

Looks like someone has come up with a nose plugin for subunit excellent! http://www.liucougar.net/blog/projects/nose-subunit In their post the author notes that subunit is not easy_installable at the moment. It will be shortly. Thanks to Tres Seaver there is a setup.py for the python component of Subunit, and he has offered to maintain that going forward. His patch is in trunk, and the next release will include a pypi upload. The next subunit release should be pretty soon too the unicode support in testtools has been overhauled thanks to Martin[gz], and so we re in much better shape on Python 2.x than we were before. Python3 for testtools is trouble free in this area because confused strings don t exist there

Robert Collins did a very nice writeup not so long ago of creating a Debian package with bzr-builddeb, but he starts from the assumption that upstream also uses bzr. As we all know, there are many projects that have their upstream sources in a VCS, but don't use bzr. Can we still get reasonable results from bzr-builddeb, including the ability to merge new upstream releases direct from the VCS, if, say, upstream is using git? Yes, we can! Of course, if you're happy with using git as the VCS for your Debian packaging already, there's probably no compelling reason for you to switch to bzr. But if you prefer to use bzr and have been frustrated at having to choose between being able to use the VCS client of your choice for your packages and being able to use your VCS client to access upstream revision history, read on. Thanks to the fine work of Jelmer Vernooij and friends, bzr-git has been usable for a while if you just want to track the default git branch. But what if you care about tracking multiple upstream branches? Enter bzr git-import, which we'll use to set up a bzr repo from scratch for our cifs-utils package, with a full import of all the branches of upstream's git tree. First we do some work to set up a shared bzr repository. As long as we're going to the effort of mirroring all the git branches, we probably want to publish them for other people to use, so we make sure our bzr repository is reasonably configured for sharing data between branches:

$ bzr init-repo --no-trees cifs-utils.server
Shared repository (format: 2a)
Location:
  shared repository: cifs-utils.server
$ cd cifs-utils.server

Then we use bzr git-import, provided by the bzr-git plugin, to do a full import of the upstream branches into a cleverly named subdirectory:

$ bzr git-import git://git.samba.org/cifs-utils.git upstream
[/                   ] Counting objects: 181, done.

Now we create our own local 'trunk' for the package, by branching from the upstream tag matching the release version we're going to package.

$ bzr branch -r tag:cifs-utils-4.0rc1 upstream/HEAD trunk
Branched 42 revision(s).

Do a little more prep work for future branches...

$ mkdir branches

And that's it. Now we have a bzr repo locally that we can push out to our hosting server of choice (N.B.: we could do this all directly on the server, but alioth doesn't currently have bzr-git installed):

$ rsync -az . alioth.debian.org:/bzr/pkg-samba/cifs-utils/
$

Now that we have a bzr repository, it's time to get ourselves a working directory and do some packaging. We're probably going to work with multiple branches locally as well, so we create another shared repository for local use, and get a copy of the trunk branch we created before.

$ bzr init-repo cifs-utils
Shared repository with trees (format: 2a)
Location:
  shared repository: cifs-utils
$ cd cifs-utils
$ bzr co bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/trunk

Now we grab the upstream tarball, and whip up some quick packaging with (what else?) debhelper 7:

$ wget ftp://ftp.samba.org/pub/samba/cifs-utils/cifs-utils-4.0rc1.tar.bz2
$ ln -s cifs-utils-4.0rc1.tar.bz2 cifs-utils_4.0~rc1.orig.tar.bz2
$ cd trunk
$ mkdir -p debian/source
$ echo '3.0 (quilt)' > debian/source/format
$ echo 7 > debian/compat
$ cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
$ dch --package cifs-utils --versio 4.0~rc1-1 --create 'Initial package'

Create debian/control by hand, generate an initial source package, and import it into bzr with bzr-builddeb.

$ debuild -uc -us -S -i
$ rm -r debian
$ bzr import-dsc ../*.dsc
$

Since we want to continue tracking upstream development, we need to periodically sync our bzr import of the git tree.

$ bzr git-import git://git.samba.org/cifs-utils.git bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/upstream
$

And when we find a new upstream version in that import that we want to package, bzr-builddeb makes this a snap, too.

$ cd cifs-utils
$ wget ftp://ftp.samba.org/pub/samba/cifs-utils/cifs-utils-$version.tar.bz2
$ cd trunk
$ bzr merge-upstream --v3 --version $version_with_epoch \
    ../cifs-utils-$version.tar.bz2 -r tag:cifs-utils-$version \
    bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/upstream/HEAD
No distribution specified, and no changelog, assuming 'debian'
Committing to: /tmp/tmpbzOVMs/upstream/
Committed revision 2.
All changes applied successfully.
The new upstream version has been imported.
You should now review the changes and then commit.
$ bzr diff
<review changes>
$ bzr commit -m "merge upstream $version"
Committing to: bzr+ssh://bzr.debian.org/bzr/pkg-samba/cifs-utils/trunk
modified debian/changelog
Committed revision 50.
$

And that's it! It's exciting to see signs of real interoperability between DVCSes at last. Thanks to everyone who's helped make it possible!

Reprap generation 20 or so + proprietary objects with embedded viruses. Real ones. (Consider what you can do in postscript )

There s a test code maintenance issue I ve been grappling with, and watching others grapple with for a while now. I ve blogged about some infrastructural things related to it before, but now I think its time to talk about the problem itself. The problem shows up as soon as you start writing setUp functions, or custom assertThing functions. And the problem is where do you put this code?

Free network services A discussion session led by Bradley Kuhn, Mako & Matt Lee : Libre.fm encouraged last.fm to write an API so they didn t need to screen scrape; outcome of the network services story still unknown netbooks without local productivity apps might now work, most users of network office apps are using them because of collaboration. We have a replacement for twitter status.net, distributed system, but nothing like facebook [yet?]. Bradley says like the original GNU problem, just start writing secure peer to peer network services to offer the things that are currently proprietary. There is perhaps a lack of an architectural vision for replacing these proprietary things: folk are asking how we will replace the cloud aspects of facebook etc tagging photos and other stuff around the web, while not using hosted-by-other-people-services. I stopped at this point to switch sessions the rooms were not in sync session time wise. Mentoring in free software Leslie Hawthorne: Projector not working, so Leslie carried on a discussion carried on from the previous talk about the use of sexual themes in promoting projects/talk content and the like. This is almost certainly best covered by watching the video. A few themes from it though:

• for anyone considering joining a community, they are assessing whether that community is people like us and for many people, including both women *and* men, blatant sexuality, isn t something that fits the people like us assessment. Note that this is in addition to offensive and inappropriate aspects of the issue.
• respect is a key element here: respect your community, respect potential contributors, and don t endorse (even silently) disrespectful behaviour
• Codes of conduct might be a good idea
• The lack of support in the community has for at least one project led to a complete loss of the women contributors to that project and they are still largely lacking many years later.

We then got Leslies actual talk. Sadly I missed the start of it I was outside organising security guards because we had (and boy it was ironic) a very loud, confrontational guy at the front who was replying to every statement and the tone in the room had gotten to the point that a fight was brewing. From where I got back:

• Check your tone
• help people be productive in your community
• cultivate creativity
• know yourself
• do not get caught up in perfectionism
• communicate both big stuff, but also just take the time to talk how are you going, etc.
• Share your mistakes
• Guide don t order
• Recognition = Retention
• Recognition = Delegation its ok to let other people be responsible for stuff
• http://bit.ly/MentorGuide
• http://bit.ly/MentoringArticle

Chris Ball, Hanna Wallach, Erinn Clark and Denise Paolucci Recruiting/retaining women in free software projects. Not a unique problem to women things that make it better for women can also increase the recruitment and retention of men. Make a lack of diversity a bug; provide onramps small easy bugs in the bug tracker (tagged as such), have a dedicated womens sub project and permit [well behaved ] men in there helps build connections into the rest of the project. Make it clear that mistakes are ok. On retention recognise first patches, first commits in newsletters and the like. Call out big things or long wanted features by the person that helped. Regular discussion of patches and fixes rather than just the changelog. CMU did a study on undergrad women participation in CS : Lack of confidence preceeds lack of interest/partipation . Engagement with what they are doing is a key thing too. Women are consistently undervaluing their worth to the free software community . Its the personal touch that seems to make a huge difference . More projects should do a code of conduct kudos to Ubuntu for doing it Chris Ball. I found the mentoring and women-in-free-software talks to have extremely similar themes which is perhaps confirmation or something but it wasn t surprising to me. They were both really good talks though! And thats my coverage of LibrePlanet I m catching a plane after lunch . Its a good low-key conference, and well put together.

John Gilmore keynote What do we do next, having produced a free software system for our computers? Perhaps we should aim at Windows? Wine + an extended ndiswrapper to run other hardware drivers + a better system administration interface/resources/manuals. However that means knowing a lot about windows internals something that open source developers don t seem to want to do. We shouldn t just carry on tweaking its not inspiring; whats our stretch goal? Discussion followed reactos, continue integrating software and people with a goal of achieving really close integration: software as human rights issue! Desktop paradigm needs to be replaced : need to move away from a document based desktop to a device based desktop. Concern about the goal of running binary drivers for hardware: encourages manufacturers to sell hardware w/out specs; we shouldn t encourage the idea that that is ok. Lots of concern about cloning, lots of concern about what will bring more freedom to users, and what it will take to have a compelling vision to inspire 50000 free software hackers. Free software in cars lots of safety issues in .e.g brake controllers, accelerators. Eben Moglen We re at the inflection point of free software because any large scale global projects these days are not feasible without free software. Claims that doing something that scales from tiny to huge environment requires us A claim I would (sadly) dispute. Lots of incoming and remaining challenges. Entirely clear that the patent systems relationship to technology is pathological and dangerous that I agree with! Patent muggings are a problem patent holders are unhappy with patents granted to other people . Patent pools are helping slowly as they grow. Companies which don t care about the freedom aspect of GPLv3 are adopting it because of the patent protection aspects. Patent system is at the head of the list of causes-of-bad-things affecting free software. SFLC is building coalitions outside the core community to protect the interests of the free software community. We are starting to be taken for granted at the high end of mgmt in companies that build on free software. We face a problem in the erosion of privacy. We need to build a stack, running on commodity hardware that runs federated services rather than folk needing centralised services. Marina Zhurakhinskaya on GNOME Shell: Integrates old and new ideas in an overall comprehensive design. Marina ran through the various goals of the shell growing with users, being delightful, starting simply so new users are not overwhelmed. The activities screen looks pretty nice The workspace rearrangement UI is really good. The notifications thing is interesting; you can respond to a chat message in-line in the notification. Richard Stallman on Software as a Service he presented verbally the case made in the paper. Some key quotes All your data on a server is equivalent to total spyware I think this is a worst-case analogy; it suggests that you can never trust another party: kindof a sad state of paranoia to assume that all network servers are always out to get you all the time. And I have to ask should we get rid of Savannah then (because all the data is stored there) the argument for why Savannah is not SaaS is not convincing: its just file storage, so what makes it different to e.g. Ubuntu One? If there is a server and only a little bit of it is SaaS, perhaps just say don t worry about it because that little bit is often the hardest bit to replace. Lets write systems for collaborative word process that don t involve a central server abiword w/the sharing plugin ? RMS seems to be claiming that someone else sysadmining a server for you is better than someone else sysadmining a time-shared server for you: I don t actually see the difference, unless you re also asserting that you ll always have root over your own machine . The argument seems very fuzzy and unclear to me as to why there is really a greater risk in particular when there is a commercial relationship with the operator (as opposed to, say, an advertising supported relationship).

GNU Hackers meetups are a face to face meeting to balance the online collaboration that GNU maintainers and contributors do all the time. These are a recent (since 2007) thing, and are having a positive effect within GNU and the FSF. The LibrePlanet 2010 GNU Hackers meetup runs concurrent with the first day of LibrePlanet. We started with some project updates:

• SipWitch a project to do discovery of SIP endpoints and setup encryption etc. This looks quite interesting, and is looking for contributors.
• Bazaar I presented an update on where Bazaar is at and what we re focusing on now and in the future:
  • short term: merging and collaboration:
    • merge behaviour
    • conflict behaviour
    • develop a rebase that can combine unrelated branches
    • looms to be polished, or pipelines extended something to manage long-standing patches for distributions, or other environments that need long lived patch sets.
  • long term
    • continuing optimisation of network and local perf
    • meta-branch operations mirror collections of branches,
    • work with many branches at once (many branches in one dir (a-la git, hopefully less confusing)
    • easier get up and go for new contributors
  • now and forever
    • keep fostering community growth
    • we re aiming for negative bug growth- get on top and stay there

Felipe Sanches presented his list of things that should be on the high priority project list:

• accessibility since 1st boot
• reconfigurable hardware development (FPGA tools) this is particularly relevant for handling e.g. wifi cards that have a FPGA in the card, so we can replace the non-free microcode.
• nonfree firmware issue

lunch John Eaton on Octave. John compared the octave contributors 30 or so over the years, and never more than 2 at a time. The Proprietary product Matlab that Octave is very similar to has 2000 staff working at the company producing it. Users seem to expect the two products to be equivalent, and are disappointed that Octave is less capable, and that the community is not as able to do the sort of support that a commercial organisation might have done. Octave would like to gain some more developers and be able to educe users more effectively convert more to become developers. Rob Myers, the chief GNU webmaster gave a description of his role: The webmasters deal with adding new content, dealing with mail to webmaster@, which can be queries for the GNU project, random questions about CDs, and an endless flood of spam. The webmasters project is run as a free software project the site is in CVS (yes CVS), visible on Savannah. Templates could be made nicer and perhaps move to a CMS. Aubrey Jaffer on cross platform. There is a thing called Water which is meant to replace all the different languages used in web apps generates html, css, alters the DOM, does what you d do with javascript. So there is a Water -> backend translator that outputs Java for servers, C# for windows, and so on. (I think, this wasn t entirely clear). He went on to talk about many of the internals of a thing called Schlep which is used as a compiler to get scheme code running in C/C#/Java so as to make it available to Water backends in different environments. Matt Lee spoke about GNU FM GNU FM is a free last.fm site. The site is running at http://libre.fm/. 24ish devs, but stalle after 6 months whats next? Matt has started GNU Social to build a communication framework for GNU projects to talk to each other e.g. for each GNU FM site to communicate on the back end, with a particular focus on doing social functionality groups, friendships, personal info. The wiki page needs ideas! GNU advisory board discussion too much to capture, but focused GNU wide issues things like how projects get contributors, contributions, coordination. Teams were a big discussion point, bug trackers how to coordinate teams followed up of that, and there is s GNU Source Release Collection project to do coordinated releases of GNU software that are all known to work together.

Not all the videos are there yet, but they are starting to show up . Yay. See http://mirror.internode.on.net/pub/linux.conf.au/2010/index.html or your local LA mirror.

Dell has been offering Ubuntu on selected models for a while. I had however nearly given up hope on being able to buy one, because they hadn t started doing that in Australia. I am very glad to see this has changed though check out their notebook page. Not all models yet, but a reasonable number have Ubuntu as an option. Yay!

So, we wanted to move a Hudson CI server at Canonical from using chroots to VM s (for better isolation and security), and there is this great product Ubuntu Enterprise Cloud (UEC basically Eucalyptus). To do this I needed to make some changes to the Hudson EC2 plugin and thats where the fun starts. While I focus on getting Hudson up and running with UEC in this post, folk generally interested in the differences between UEC and EC2, or getting a single-machine UEC instance up for testing should also find this useful. Firstly, getting a test UEC instance installed was a little tricky I only had one machine to deploy it on, and this is an unusual configuration. Nicely though, it all worked, once a few initial bugs and misconfiguration items got fixed up. I wrote up the crux of the outcome on the Ubuntu community help wiki. See 1 Physical system . The particular trap to watch out for seems to be that this configuration is not well tested, so the installation scripts have a hard time getting it right. I haven t tried to make it play nice with Network Manager in the loop, but I m pretty sure that that can be done via interface aliasing or something similar. Secondly I needed to find out what was different between EC2 and UEC (Note that I was running on Karmic (Ubuntu 9.10) so things could be different in Lucid). I couldn t find a simple description of this, so this list may be incomplete:

1. UEC runs an old version of the EC2 API. This is because it hasn t implemented everything in the new API versions yet.
2. UEC defaults to port 8773, not port 80 (for both the EC2 and S3 API s)
3. The EC2 and S3 API s are rooted differently: at AWS they are at /, for UEC they are at /services/Eucalyptus and /services/Walrus
4. UEC doesn t supply a SSL API port as far as I can tell.
5. DescribeImages has something wonky with it.

So the next step then is to modify the Hudson EC2 plugin to support these differences. Fortunately it is in Java, and the Java community has already updated the various libraries (jets3t and typica) to support UEC I just needed to write a UI for the differences and pass the info down the various code paths. Kohsuke has let me land this now even though it has an average UI (in rev 27366), and I m going to make the UI better now by consolidating all the little aspects into a couple of URL s. Folk comfortable with building their own .hpi can get this now by svn updating and rebuilding the ec2 plugin. We ve also filed another bug asking for a single API call to establish the endpoints, so that its even easier for users to set this up. Finally, and this isn t a UEC difference, I needed to modify the Hudson EC2 plugin to work with the ubuntu user rather than root, as Ubuntu AMI s ship with root disabled (as all Ubuntu installs do). I chose to have Hudson reenable root, rather than making everything work without root, because the current code paths assume they can scp things as root, so this was less disruptive. With all that done, its now possible to configure up a Hudson instance testing via UEC nodes. Here s how:

1. Install UEC and make sure you can run up instances using euca-run-instances, ssh into them and that networking works for you. Make sure you have installed at least one image (EMI aka AMI) to run tests on. I used the vanilla in-store UEC Karmic images.
2. Install Hudson and the EC2 plugin (you ll need to build your own until a new release (1.6) is made).
3. Go to /configure and near the bottom click on Add a new cloud and choose Amazon EC2.
4. Look in ~/.euca/eucarc, or in the zip file that the UEC admin web page lets you download, to get at your credentials. Fill in the Access Key and Secret Access key fields accordingly. You can put in the private key (UEC holds onto the public half) that you want to use, or (once the connection is fully setup) use the Generate Key button to have a dedicated Hudson key created. I like to use one that I can ssh into to look at a live node YMMV. (Or you could add a user and as many keys as you want in the init script more on that in a second).
5. Click on Advanced, this will give you a bunch of details like EC2 Endpoint hostname . Fill these out.
6. Sensible values for a default UEC install are: 8773 for both ports, /services/Eucalyptus and /services/Walrus for the base URLs, and SSL turned off. (Note that the online help tells you this as well).
7. Set an instance cap, unless you truely have unlimited machines. E.g. 5, to run 5 VMs at a time.
8. Click on Test Connection it should pretty much instantly say Success .
9. Thats the Cloud itself configured, now we configure VM s that Hudson is willing to start. Click on Add right above the List of AMIs to be launched as slaves text.
10. Fill out the AMI with your EMI e.g. emi-E027107D is the Ubuntu 9.10 image I used.
11. for remote FS root, just put /hudson or something, unless you have a preseeded area (e.g. with a shared bzr repo or something) inside your image.
12. For description describe the intent of the image e.g. DB test environment
13. For the labels put one or more tags that you will use to tell test jobs they should run on this instance. They can be the same as labels on physical machines it will act as an overflow buffer. If no physical machines exist, a VM will be spawned when needed. For testing I put euca
14. For the init script, its a little more complex. You need to configure up java so that hudson itself can run:

    cat >> /etc/apt/sources.list << EOF
    deb http://archive.ubuntu.com/ubuntu/ karmic multiverse
    deb http://archive.ubuntu.com/ubuntu/ karmic-updates multiverse
    deb http://archive.ubuntu.com/ubuntu/ karmic-security multiverse
    EOF
    export http_proxy=http://192.168.1.1:8080/
    export DEBIAN_FRONTEND=noninteractive
    apt-get update
    echo "buildd shared/accepted-sun-dlj-v1-1 boolean true"   debconf-set-selections
    apt-get install -y -f sun-java6-jre
    

    Note that I have included my local HTTP proxy there just remove that line if you don t have one.
15. Click on Advanced, to get at the less-common options.
16. For remote user, put ubuntu , and for root command prefix put sudo .
17. For number of executors, you are essentially choosing the number of CPU s that the instance will request. E.g. putting 20 will ask for an extra-large high-cpu model machine when it deploys. This will then show up as 20 workers on the same machine.
18. Click save
19. Now, when you add a job a new option in the job configuration will appear tie this job to a node . Select one of the label(s) you put in for the AMI, and running the job will cause that instance to start up if its not already available.

Note that Hudson will try to use java from s3 if you don t install it, but that won t work right for a few reasons I ll be filing an issue in the Hudson tracker about it, as thats a bit of unusual structure in the existing code that I m happier leaving well enough alone .

Looking at using google apps for my home email, as I want to be able to have my home machines totally turned off from time to time. Found this interesting gem in the sign up agreement (which I have not yet agreed to ):

11. PR. Customer agrees not to issue any public announcement regarding the existence or content of this Agreement without Google s prior written approval. Google may (i) include Customer s Brand Features in presentations, marketing materials, and customer lists (which includes, without limitation, customer lists posted on Google s web sites and screen shots of Customer s implementation of the Service) and (ii) issue a public announcement regarding the existence or content of this Agreement. Upon Customer s request, Google will furnish Customer with a sample of such usage or announcement.

This is rather asymmetrical: If I agree to the sign up page, I cannot say I am using google apps , but google can say Robert is using google apps . While I can appreciate not wanting to be dissed on if something goes wrong, this is very much not open! A couple of implications: Everyone seeking support for google apps in the apps forums is probably in violation of the sign up agreement; we can assume that anyone having a terrible experience has been squelched under this agreement. Le sigh.